fix: fail Deployment tracking fast on ReplicaSet pod-create errors - #399
Open
Sharvash wants to merge 2 commits into
Open
fix: fail Deployment tracking fast on ReplicaSet pod-create errors#399Sharvash wants to merge 2 commits into
Sharvash wants to merge 2 commits into
Conversation
Member
|
Thanks! But the main branch is kinda freezed right now, the development is mostly in the "1" branch. A big portion of the kubedog codebase is a mess, so I don't want to bring features to both branches |
Contributor
Author
My plan:
Should I go ahead? |
Member
|
yes |
Sharvash
added a commit
to Sharvash/kubedog
that referenced
this pull request
Jul 24, 2026
Problem: during a Deployment rollout, when the child ReplicaSet cannot create pods (for example `FailedCreate ... exceeded quota`), kubedog did not surface the failure and tracking hung until timeout. Root cause: the ReplicaSet controller emits these Warning events with InvolvedObject set to the ReplicaSet, not the Deployment. The deployment tracker only ran an event informer for the Deployment object itself, so ReplicaSet-scoped failures were never observed. StatefulSet and DaemonSet are unaffected because their controllers emit FailedCreate on the controller object kubedog already watches. The fix: start an additional event informer for each new ReplicaSet as soon as it is discovered (tracked per RS name, so a new ReplicaSet appearing mid-rollout gets its own informer too), feeding the same resourceFailed channel the Deployment tracker already reacts to. Any "Failed*" reason (including FailedCreate: exceeded quota) now interrupts tracking, consistent with StatefulSet/DaemonSet behavior. Honors KUBEDOG_DISABLE_EVENTS=1. It does NOT change the event package, the failure-counting semantics of dyntracker, or add a new immediate-abort bypass; it only widens what the Deployment tracker observes. Verified with go build ./... and go vet ./... on this branch. The behavior fix was reproduced against the main branch (PR werf#399), whose deployment tracker differs from this one only in import paths, using a kind cluster with a ResourceQuota blocking pod creation: before, Deployment tracking hung for the full timeout; after, it failed within seconds with the FailedCreate reason. Closes werf#398 Refs werf#216, werf#363 Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
Sharvash
force-pushed
the
fix/398-replicaset-quota-events
branch
from
July 24, 2026 11:30
f5c94c5 to
405a276
Compare
Sharvash
marked this pull request as draft
July 24, 2026 13:28
Read FailedCreate from the durable ReplicaSetReplicaFailure condition instead of transient Warning events, so failures that predate informer startup terminate tracking immediately. Only consider ReplicaSets controlled by the tracked Deployment to avoid selector collisions. Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
Force Failed payloads to set IsFailed and clear IsReady, while ignoring lagging FailedCreate conditions after the Deployment becomes ready. Deduplicate failures per ReplicaSet incarnation, rearm reporting after condition recovery or Deployment readiness, ignore stale events after deletion, and remove unused failure state. Re-evaluate the new ReplicaSet once a ReplicaSet is deleted: the deletion may promote a template-equivalent successor that is already failing, and its failure would otherwise never be reported. Deliver ReplicaSet events over unbuffered channels. Additions, modifications and deletions come from a single informer goroutine but fan out into three channels, so buffering let the tracker pick a newer modification before an older addition and overwrite a fresh snapshot with a stale one, either failing a rollout that had already recovered or losing a failure that recurred. Qualify reported failures with a FailureMode: failures coming from events stay counted against the allowed failures count, while a durable ReplicaSet ReplicaFailure condition is reported as fatal. The dynamic readiness tracker marks the resource state of a fatal failure as failed, so a single ReplicaSet pod-create error terminates tracking instead of being swallowed by the default failures allowance. Fail the readiness task on a failed resource state only for fail modes that are supposed to fail the deploy process, keeping IgnoreAndContinueDeployProcess unaffected. Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
Sharvash
force-pushed
the
fix/398-replicaset-quota-events
branch
from
August 8, 2026 13:55
405a276 to
e995853
Compare
Sharvash
marked this pull request as ready for review
August 8, 2026 13:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes the Deployment tracker fail fast on ReplicaSet pod-create errors such as
FailedCreate ... exceeded quota, instead of hanging until the timeout.The ReplicaSet controller reports pod-create failures on the ReplicaSet — a
FailedCreateWarning event and a durableReplicaSetReplicaFailurecondition — while the deployment tracker only watched the Deployment object and its events, so these failures were never observed. StatefulSet and DaemonSet are unaffected because their controllers reportFailedCreateon the object kubedog already watches.fix: fail Deployment tracking on ReplicaSet pod-create errors
FailedCreatefrom the durableReplicaSetReplicaFailurecondition of the Deployment's ReplicaSets instead of transient Warning events: the condition is set while pod creation keeps failing and removed once it succeeds, so failures that predate informer startup terminate tracking immediately and no event baseline is needed. Also works withKUBEDOG_DISABLE_EVENTS=1.fix: handle ReplicaSet failure transitions consistently
IsFailedand clearIsReady; laggingFailedCreateconditions after the Deployment becomes ready are ignored.FailedCreatemessages carry volatile quota numbers — and rearmed after condition recovery or Deployment readiness, so a failure that recurs is reported again.FailureMode: failures coming from events stay counted against the allowed-failures budget, while a durable ReplicaSetReplicaFailurecondition is fatal — the dynamic readiness tracker marks the resource failed, so a single pod-create error terminates tracking instead of being swallowed by the default failures allowance. Fail modes that are not supposed to fail the deploy process (IgnoreAndContinueDeployProcess) are unaffected.Verification:
go build ./...,go vet ./...,go test ./..., including new lifecycle tests for the failure transitions (dedupe, rearm, stale events, successor promotion) over fake clientsets,-raceruns on the deployment/replicaset packages, and a regression test that proves the informer goroutine is released after cleanup. Originally reproduced on a kind cluster (namespace withResourceQuota pods: 0, 1-replica Deployment): tracking hung for the full timeout before the fix and fails within seconds with theFailedCreatequota message after it.Closes #398
Refs #216, #363